fix(archive): re-zip fresh on an unusable resume candidate instead of hard-failing #12
+131
−17
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… hard-failing A scratch-purged, zero-byte (partial-write), or corrupt resume zip is fully recoverable — the source on /depot is intact, so the run should just re-zip from it. Previously reconstruct_from_zip's failure hard-exited (sys.exit 1) and emailed a FAILED alert, which burned the resume watchdog's retry budget on a self-healing condition and stranded the experiment with nothing on Fortress (X0H_Soy: 5.5 TB never archived; watchdog attempts maxed at 3). - find_existing_zip: skip candidates that don't exist or are zero-byte, so a purged/partial stub is never offered as a resume source. - try_reconstruct(existing_zip, runner): new helper — returns the rebuilt tuple, or None (after removing the bad zip) when the candidate is unusable, so the caller falls through to a fresh make_zip_files. - main(): a None result re-zips fresh instead of exiting. The staleness guard (incomplete-snapshot protection) is unchanged. Tests: tests/test_resume_fallback.py (find_existing_zip skip-logic; try_reconstruct valid/missing/corrupt). Full suite 43 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
When a resume run finds a staged zip that turns out to be unusable —
/scratchpurged it, a crash left a 0-byte stub, or it's corrupt —archive.pyused to hard-fail with an alert email andsys.exit(1). That burns the resume watchdog's retry budget on a self-healing condition and can strand an experiment with nothing on Fortress (this is what happened toX0H_Soy).This change makes that path self-heal: an unusable resume candidate is removed and the run re-zips fresh from the (intact)
/depotsource, which is correct and complete.How
find_existing_zipnow skips candidates that don't exist or are zero-byte.try_reconstruct(existing_zip, runner)wrapsreconstruct_from_zip; on any exception it logs[RESUME] Candidate unusable …, removes the bad zip, and returnsNoneso the caller falls through to a fresh zip.__main__resume block: aNoneresult clearsexisting_zipand proceeds to re-zip instead of exiting.Tests
tests/test_resume_fallback.py(+6 tests): corrupt zip, missing/purged zip, and 0-byte stub all fall back to a fresh re-zip rather than aborting. Full suite green: 43 tests, OK.Notes
Independent of the incremental/size-routing redesign — a standalone production bug fix. Rebases cleanly onto current
main(verified; #5/#6 net to ~no-op onarchive.py).Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
🤖 Generated with Claude Code